{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pima Indians Diabetes Data Set  Logistic 回归\n",
    "\n",
    "数据说明：\n",
    "Pima Indians Diabetes Data Set（皮马印第安人糖尿病数据集） 根据现有的医疗信息预测5年内皮马印第安人糖尿病发作的概率。   \n",
    "\n",
    "数据集共9个字段: \n",
    "0列为怀孕次数；\n",
    "1列为口服葡萄糖耐量试验中2小时后的血浆葡萄糖浓度；\n",
    "2列为舒张压（单位:mm Hg）\n",
    "3列为三头肌皮褶厚度（单位：mm）\n",
    "4列为餐后血清胰岛素（单位:mm）\n",
    "5列为体重指数（体重（公斤）/ 身高（米）^2）\n",
    "6列为糖尿病家系作用\n",
    "7列为年龄\n",
    "8列为分类变量（0或1）\n",
    "\n",
    "数据链接：https://archive.ics.uci.edu/ml/datasets/Pima+Indians+Diabetes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import必要的工具包，用于文件读取／特征编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据文件路径和文件名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnants</th>\n",
       "      <th>Plasma_glucose_concentration</th>\n",
       "      <th>blood_pressure</th>\n",
       "      <th>Triceps_skin_fold_thickness</th>\n",
       "      <th>serum_insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>Diabetes_pedigree_function</th>\n",
       "      <th>Age</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.639947</td>\n",
       "      <td>0.866045</td>\n",
       "      <td>-0.031990</td>\n",
       "      <td>0.670643</td>\n",
       "      <td>-0.181541</td>\n",
       "      <td>0.166619</td>\n",
       "      <td>0.468492</td>\n",
       "      <td>1.425995</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.844885</td>\n",
       "      <td>-1.205066</td>\n",
       "      <td>-0.528319</td>\n",
       "      <td>-0.012301</td>\n",
       "      <td>-0.181541</td>\n",
       "      <td>-0.852200</td>\n",
       "      <td>-0.365061</td>\n",
       "      <td>-0.190672</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.233880</td>\n",
       "      <td>2.016662</td>\n",
       "      <td>-0.693761</td>\n",
       "      <td>-0.012301</td>\n",
       "      <td>-0.181541</td>\n",
       "      <td>-1.332500</td>\n",
       "      <td>0.604397</td>\n",
       "      <td>-0.105584</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.844885</td>\n",
       "      <td>-1.073567</td>\n",
       "      <td>-0.528319</td>\n",
       "      <td>-0.695245</td>\n",
       "      <td>-0.540642</td>\n",
       "      <td>-0.633881</td>\n",
       "      <td>-0.920763</td>\n",
       "      <td>-1.041549</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.141852</td>\n",
       "      <td>0.504422</td>\n",
       "      <td>-2.679076</td>\n",
       "      <td>0.670643</td>\n",
       "      <td>0.316566</td>\n",
       "      <td>1.549303</td>\n",
       "      <td>5.484909</td>\n",
       "      <td>-0.020496</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pregnants  Plasma_glucose_concentration  blood_pressure  \\\n",
       "0   0.639947                      0.866045       -0.031990   \n",
       "1  -0.844885                     -1.205066       -0.528319   \n",
       "2   1.233880                      2.016662       -0.693761   \n",
       "3  -0.844885                     -1.073567       -0.528319   \n",
       "4  -1.141852                      0.504422       -2.679076   \n",
       "\n",
       "   Triceps_skin_fold_thickness  serum_insulin       BMI  \\\n",
       "0                     0.670643      -0.181541  0.166619   \n",
       "1                    -0.012301      -0.181541 -0.852200   \n",
       "2                    -0.012301      -0.181541 -1.332500   \n",
       "3                    -0.695245      -0.540642 -0.633881   \n",
       "4                     0.670643       0.316566  1.549303   \n",
       "\n",
       "   Diabetes_pedigree_function       Age  Target  \n",
       "0                    0.468492  1.425995       1  \n",
       "1                   -0.365061 -0.190672       0  \n",
       "2                    0.604397 -0.105584       1  \n",
       "3                   -0.920763 -1.041549       0  \n",
       "4                    5.484909 -0.020496       1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#input data\n",
    "data = pd.read_csv(\"FE_pima-indians-diabetes.csv\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "labels = data['Target']\n",
    "features = data.drop([\"Target\"], axis=1)\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 划分训练集和测试集\n",
    "X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.3, random_state=8)\n",
    "\n",
    "# 初始化特征的标准化器 \n",
    "ss_X = StandardScaler()\n",
    "# 对训练和测试数据的特征进行标准化处理 \n",
    "X_train = ss_X.fit_transform(X_train) \n",
    "X_test = ss_X.transform(X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[1e-05, 0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "                     class_weight=None, cv=3, dual=False, fit_intercept=True,\n",
       "                     intercept_scaling=1.0, l1_ratios=None, max_iter=100,\n",
       "                     multi_class='ovr', n_jobs=None, penalty='l2',\n",
       "                     random_state=None, refit=True, scoring='neg_log_loss',\n",
       "                     solver='lbfgs', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " \n",
    "from sklearn.linear_model import LogisticRegressionCV \n",
    "# 设置正则项系数搜索空间\n",
    "Cs = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]\n",
    "# 创建L1和L2正则的logistic回归估计器\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 3, scoring='neg_log_loss', penalty='l1', solver='liblinear', multi_class='ovr')\n",
    "lrcv_L2 = LogisticRegressionCV(Cs=Cs, cv = 3, scoring='neg_log_loss', penalty='l2', solver='lbfgs', multi_class='ovr')\n",
    "# 训练估计器 \n",
    "lrcv_L1.fit(X_train, y_train) \n",
    "lrcv_L2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LogisticRegressionCV+L1 scores:\n",
      "{1: array([[-0.69314718, -0.69314718, -0.69314718, -0.69314718, -0.46127608,\n",
      "        -0.43970129, -0.44004384, -0.4401154 , -0.44012488],\n",
      "       [-0.69314718, -0.69314718, -0.69314718, -0.69314718, -0.52176369,\n",
      "        -0.5364972 , -0.54389467, -0.5447167 , -0.54480917],\n",
      "       [-0.69314718, -0.69314718, -0.69314718, -0.69314718, -0.48361183,\n",
      "        -0.46112096, -0.45840515, -0.45817374, -0.4581447 ]])} \n",
      "\n",
      "LogisticRegressionCV+L2 scores:\n",
      "{1: array([[-0.64475578, -0.64050488, -0.60594147, -0.50126189, -0.4447359 ,\n",
      "        -0.4399827 , -0.44009671, -0.44012051, -0.44012379],\n",
      "       [-0.64115947, -0.63756079, -0.60935178, -0.53616956, -0.52040855,\n",
      "        -0.53941099, -0.54422341, -0.54476852, -0.54482355],\n",
      "       [-0.64116294, -0.63759362, -0.6094397 , -0.52710942, -0.47146436,\n",
      "        -0.45964064, -0.45829809, -0.45815989, -0.45814688]])}\n"
     ]
    }
   ],
   "source": [
    "# 查看不同Cs下交叉验证的模型score(这里设置的是neg_log_loss) \n",
    "# 行代表交叉验证集，列代表Cs \n",
    "print(\"LogisticRegressionCV+L1 scores:\") \n",
    "print(lrcv_L1.scores_,'\\n') \n",
    "print(\"LogisticRegressionCV+L2 scores:\") \n",
    "print(lrcv_L2.scores_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEHCAYAAAC0pdErAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZwcdZ3/8dene64kM9OTY3J1hyRAgiTpIUAIR4LrCRGVQ5BDEZAFdBcWFZf94borLiyrj12vVVE5RMCDYwERFUVEkGuRhCvngiEEmCSQyUHuzPn5/VE1oTN0Mj2Z7qnp7vfz8ajHdFVXVX+SQL/nW9/6fsvcHRERkZ5iURcgIiKDkwJCRESyUkCIiEhWCggREclKASEiIlkpIEREJKuKQp7czOYB/w3EgRvd/es93v828N5wdSgw2t0bwvfOBf4lfO/f3f2WvX3WqFGjfNKkSXmsXkSk9D3zzDPr3L0x23tWqHEQZhYHXgI+CDQD84Gz3H3pHvb/B+BQdz/fzEYAC4BZgAPPAIe7+8Y9fd6sWbN8wYIFef5TiIiUNjN7xt1nZXuvkJeYZgPL3X2Fu7cBtwMn7WX/s4DbwtfHAw+6+4YwFB4E5hWwVhER6aGQAZEEXs9Ybw63vYOZTQQmA3/q67EiIlIYg6WT+kzgLnfv7MtBZnaRmS0wswUtLS0FKk1EpDwVspN6FTAhYz0VbsvmTODiHse+p8exj/Q8yN2vB66HoA9i30sVEemb9vZ2mpub2blzZ9Sl5KSmpoZUKkVlZWXOxxQyIOYDU8xsMsEX/pnAJ3ruZGbvAoYD/5ux+QHgP8xseLh+HPClAtYqItInzc3N1NXVMWnSJMws6nL2yt1Zv349zc3NTJ48OefjCnaJyd07gEsIvuyXAXe6+xIzu8rMTszY9Uzgds+4ncrdNwBXE4TMfOCqcJuIyKCwc+dORo4cOejDAcDMGDlyZJ9bOwUdB+Hu9wP399j2lR7rX93DsTcBNxWsOBGRfiqGcOi2L7UWNCCKwfa2Dn70yMtRlyH5YMYphyaZPGpY1JWIlISyD4gdbZ187+HlUZcheeAOq9/awTc+fkjUpYgMiNraWrZu3QrAvHnzeOqpp5g7dy6/+c1v8nL+sg+IkbXVvPK1D0ddhuTBp3/yNItXbYq6DJFIXH755Wzfvp3rrrsub+ccLOMgRPotnUzw0ptb2NHWp+E0IiXh/e9/P3V1dXk9Z9m3IKR0pFMNdDksXbOJwyeOiLocKSP/9uslLF29Oa/nnDa+nis/Oj2v5+wrtSCkZDSlEgAsatZlJpF8UAtCSsaY+hoa66pZqH4IGWBR/6ZfKGpBSElpSibUghDJEwWElJR0KsHLLVvZ1toRdSkiA+rYY4/l4x//OA899BCpVIoHHnig3+fUJSYpKelkIuyo3swRk9RRLaWtewwEwGOPPZb386sFISUlnQw6qhfqMpNIvykgpKSMrq9hbH0Ni5rfiroUkaKngJCSMyOZYJHuZBLpNwWElJymVIIV67axZWd71KWIFDUFhJScdCqBOyzJ88hWkXKjgJCS091RrYn7RPpHASElZ1RtNeMTNbqTSUpebW0tAM8//zxHH30006dPp6mpiTvuuCMv59c4CClJ6ZQ6qqV8DB06lFtvvZUpU6awevVqDj/8cI4//ngaGhr6dV61IKQkNaUaeGXdNjaro1rKwNSpU5kyZQoA48ePZ/To0bS0tPT7vGpBSEmakdEPccwBoyKuRkre766ANxbl95xj0/Chr/f5sKeffpq2tjYOOOCAfpegFoSUpO6Oak3cJ+VkzZo1fOpTn+InP/kJsVj/v97VgpCSNGJYFanhQ9QPIQNjH37Tz7fNmzfz4Q9/mGuuuYajjjoqL+dUC0JKVlojqqVMtLW1ccopp3DOOedw2mmn5e28CggpWelUglfXb2fTdnVUS2m78847efTRR7n55puZOXMmM2fO5Pnnn+/3eXWJSUpWUzK4xW/x6k3MOVAd1VJ6uqf7Pvvsszn77LPzfn61IKRkzUjWA5r6W2RfKSCkZDUMrWK/EUNZtEpTf4vsCwWElDSNqJZCcveoS8jZvtSqgJCSlk4meH3DDjZua4u6FCkxNTU1rF+/vihCwt1Zv349NTU1fTpOndRS0pq6B8yt2sS7pzZGXI2UklQqRXNzc16mtBgINTU1pFKpPh2jgJCSNl0BIQVSWVnJ5MmToy6joHSJSUpaYkglk0YO1ZQbIvtAASElL51qUEe1yD5QQEjJa0omWPXWDtZvbY26FJGiooCQkjcjox9CRHKngJCS1z2iWv0QIn2jgJCSV1dTyf6Nw1ioFoRInxQ0IMxsnpm9aGbLzeyKPexzupktNbMlZvaLjO2dZvZ8uNxXyDql9DUlEyxWQIj0ScHGQZhZHLgW+CDQDMw3s/vcfWnGPlOALwFz3H2jmY3OOMUOd59ZqPqkvMxIJrj3+dWs3bKT0XV9G00qUq4K2YKYDSx39xXu3gbcDpzUY58LgWvdfSOAu68tYD1SxppS4dTfakWI5KyQAZEEXs9Ybw63ZZoKTDWzJ8zsKTObl/FejZktCLefXMA6pQxMH1+PGSxq3hx1KSJFI+qpNiqAKcB7gBTwqJml3f0tYKK7rzKz/YE/mdkid38582Azuwi4CGC//fYb2MqlqAyrruCAxlpN/S3SB4VsQawCJmSsp8JtmZqB+9y93d1fAV4iCAzcfVX4cwXwCHBozw9w9+vdfZa7z2ps1Dw7sndNyYQeHiTSB4UMiPnAFDObbGZVwJlAz7uR7iVoPWBmowguOa0ws+FmVp2xfQ6wFJF+SKcSrN3Sypubd0ZdikhRKFhAuHsHcAnwALAMuNPdl5jZVWZ2YrjbA8B6M1sKPAxc7u7rgYOBBWb2Qrj965l3P4nsi3T3iGq1IkRyUtA+CHe/H7i/x7avZLx24LJwydznSSBdyNqk/EwbX0/MYOGqTXxg2pioyxEZ9DSSWsrG0KoKpoyu062uIjmK+i6m6LXvhHsugNoxwTKsMXw9OliGjYZKDawqFTOSCf78UgvujplFXY7IoKaAaN0M6/4KKx+HHRuz71OdCANjDNQ29giSHtvilQNbv/RJUyrB3c8288bmnYxLDIm6HJFBTQFROxou/kvwuqMNtrXA1jff/rn1Tdiase2NRbD1oSBYshkyokdojH5nuNSOgaEjIRYfuD+nAMGdTAALmzcpIER6oYDIVFEFiWSw9KZ9B2xdGyzb1oZBEq53h0nz/GC9ffs7j7cYDB319qWsbJe3asfCqKkQU1dRvkwbV088ZixetYnjp4+NuhyRQU0Bsa8qh8DwicHSm9atbwfItrW7B0n3tnV/DdY723Y/dvK74WM3QJ2+zPKhpjLOlNG1GjAnkgMFxECorg2WkQfsfT932Lnp7fBY8wI8fA38cA6cch1M+cDA1FvimlIJ/rhsrTqqRXqhaxeDiRkMaYDGqTD5WDjmErjokeCy089PhT/8a9BPIv2STibYsK2N1Zs0olpkbxQQg13jQXDhQzDrb+HJ78JP5sGGV6Kuqqilw6m/FzVr4j6RvVFAFIPKIfCRb8Hpt8K65XDdu2HxPVFXVbTeNbaOipipH0KkFwqIYjLtJPjsY0Gr4q5Pw32XQluWO6Rkr2oq40wdU8cijagW2SsFRLEZPhE+/TuY+wV49la44X2wdlnUVRWdplSCRas2EUwHJiLZKCCKUbwSPvBV+NQ9sH0dXP8eeObm4C4oyUk6leCt7e00b9wRdSkig5YCopgd8D747BOw39Hw688Fl5126rJJLpqSYUe1LjOJ7JECotjVjYGz7wlaFEvvgx/NheYFUVc16E0dW0tlXB3VInujgCgFsVjQJ3H+78GBm46HJ/4burqirmzQqq6I866x9XpGtcheKCBKyYTZwV1OB50AD34Ffn5aMNGgZJVOJVjUrI5qkT1RQJSaIQ3BeImPfBtefQJ+NAdWPBJ1VYNSOplg884OXtugW4VFslFAlCIzmHU+XPgnqGmAW0+Gh66Czo6oKxtUup9RrX4IkewUEKVszHS46GE49Gx47Jtw8wnw1utRVzVoTB1TR1VFTHcyieyBAqLUVQ2Dk74Pp/4Y3lwaXHJa9uuoqxoUqipiHDy2jkVqQYhkpYAoF+nT4LOPwoj94Y6z4bf/GDyPu8ylUwkWr9pEV5c6qkV6UkCUkxH7w/l/gKMvgfk3wI3vh5aXoq4qUk3JBra0drBy/baoSxEZdBQQ5aaiCo6/Bj7xP7BlDVz/N/Dcz8t2mo4ZYUe1+iFE3kkBUa6mHgeffRySh8Ov/h7uuQhat0Rd1YCbMqaW6oqY+iFEslBAlLP68XDOr+C9X4bFdwXPmVj9fNRVDajKeIxp4+tZqBaEyDsoIMpdLA5/809w3m+hoxVu/AA89cOyuuSUTiZYoo5qkXdQQEhg4jHBJacpH4TfXwG3nQnb1kdd1YBIJxNsa+tkxTp1VItkUkDI24aOgDN/AR/6T3j5T8HMsCsfj7qqgmvqfka1Ju4T2Y0CQnZnBkd+Bi74Y/As7Fs+Co98Hbo6o66sYA5oHEZNZYxFzZujLkVkUFFASHbjDoHP/BnSp8MjX4NbToTNq6OuqiAq4jGmj0+oBSHSgwJC9qy6Dj52HZz8I1j9HPxwDrz4+6irKoh0MsHiVZvpVEe1yC4KCOndzLOC1kQiCbedAb//UnDHUwlpSiXY0d7JipatUZciMmgoICQ3o6bA3/4RZn8GnvpBMH14CdHU3yLvpICQ3FXWwAn/CYecBQtuKqnbYPdvrGVoVVxTbohkUEBI3839ArTvgL/8KOpK8iYeM2aMTyggRDIoIKTvGg+Cgz8CT18HO0vn1tAZyQRLVm+io7Mr6lJEBoWCBoSZzTOzF81suZldsYd9TjezpWa2xMx+kbH9XDP7a7icW8g6ZR/MvQx2boJnfhJ1JXnTlEqws72L5eqoFgEKGBBmFgeuBT4ETAPOMrNpPfaZAnwJmOPu04HPh9tHAFcCRwKzgSvNbHihapV9kDwM9n8vPPn9knnwUDqljmqRTH0OCDOLmVl9DrvOBpa7+wp3bwNuB07qsc+FwLXuvhHA3deG248HHnT3DeF7DwLz+lqrFNixX4Rta+H5n0VdSV5MHjmM2uoKFqsfQgTIMSDM7BdmVm9mw4DFwFIzu7yXw5LA6xnrzeG2TFOBqWb2hJk9ZWbz+nCsRG3SXEgdAU/8N3R2RF1Nv8VixvTx9WpBiIRybUFMc/fNwMnA74DJwKfy8PkVwBTgPcBZwA1m1pDrwWZ2kZktMLMFLS0teShH+sQsaEW89RosvjvqavKiKZVg6ZrNtKujWiTngKg0s0qCgLjP3duB3uYkWAVMyFhPhdsyNXefz91fAV4iCIxcjsXdr3f3We4+q7GxMcc/iuTVlONh9HR4/FvQVfxfqjOSCdo6uvjrm+qoFsk1IK4DVgLDgEfNbCLQ2/2N84EpZjbZzKqAM4H7euxzL0HrATMbRXDJaQXwAHCcmQ0PO6ePC7fJYBOLwbGXQcv/wYv3R11Nv2nqb5G35RQQ7v5dd0+6+wkeeBV4by/HdACXEHyxLwPudPclZnaVmZ0Y7vYAsN7MlgIPA5e7+3p33wBcTRAy84Grwm0yGE07GYZPgse+WfRPops4Yih1NRXqhxAh907qz4Wd1GZmPzazZ4H39Xacu9/v7lPd/QB3vybc9hV3vy987e5+mbtPc/e0u9+ecexN7n5guJTOzfalKF4Bcz4Pq5+FV/4cdTX9EgtHVOtOJpHcLzGdH3ZSHwcMJ+ig/nrBqpLiM/MTUDs2aEUUuaZUgmVrttDWUfx9KiL9kWtAWPjzBOCn7r4kY5sIVFTDMf8ArzwKzQuirqZf0qkEbZ1dvPTmlqhLEYlUrgHxjJn9gSAgHjCzOkC/XsnuDj8PhgyHx74VdSX90j31tybuk3KXa0D8LXAFcIS7bweqgE8XrCopTtW1cORn4cXfwptLo65mn+03Yij16qgWyfkupi6CsQj/YmbfAI5x94UFrUyK0+yLoHIYPP7tqCvZZ2ZGU6pBt7pK2cv1LqavA58DlobLpWb2H4UsTIrU0BFwxPmw+C7Y8ErU1eyzdCrBi29sobWjM+pSRCKT6yWmE4APhree3kQwcd5HCleWFLWjLoZYBTz53agr2WfpZIL2TufFN9RRLeWrL7O5Zs6RlMh3IVJC6sfBzE/Ccz+DLW9EXc0+0TOqRXIPiK8Bz5nZzWZ2C/AMcE3hypKiN+dz0NUB//v9qCvZJ6nhQxg+tFID5qSs5dpJfRtwFHAPcDdwtLvfUcjCpMiNmAwzToX5N8H24pslxcyYkUyoBSFlba8BYWaHdS/AOILZV5uB8eE2kT2b+wVo3wZP3xB1JfukKZXgpTe3sLNdHdVSnip6eX9v8yY4OczHJGVszHQ46AT4yw/h6IuDcRJFJJ1soKPLWbZmM4fupyfeSvnZa0C4+15nbBXp1dzL4McfgGduhmMuibqaPul+RvXiVZsUEFKWemtBAGBmH8uyeROwKOM50iLvNOEImHRs0Fk9+8JgzqYiMT5Rw8hhVeqHkLLVl6k2bgQ+GS43AP8PeMLM8vHoUSllx34RtqyBF26LupI+MTPSqYTmZJKylWtAVAAHu/up7n4qMI2gD+JIgqAQ2bP93wPjD4PHvwOdHVFX0yfpZIK/rt3KjjZ1VEv5yTUgJrj7mxnra8NtG4D2/JclJcUsaEVsfAWW3ht1NX2STibo7HKWruntCbsipSfXgHjEzH5jZuea2bkEz5Z+xMyGAZrRTHp30AnQ+K5gKvAieizprmdUN+s/cyk/uQbExcBPgJnhcgtwsbtv051OkpNYLBgXsXYJvPRA1NXkbEx9NaNqq1m0Si0IKT+5jqR24HHgT8BDwKPhNpHczTgVGvaDx75RNK2IYOrvhKb+lrKU63TfpwNPA6cBpwN/MbPTClmYlKB4JRxzKTTPh5WPR11NztLJBMvXbmV7W3F1sIv0V66XmL5M8DS5c939HGA28K+FK0tK1qFnw7DR8HjxPJY0nUzQ5bB0tS4zSXnJNSBiPQbEre/DsSJvqxwSTLvx8p9g1bNRV5OT7hHVGjAn5SbXL/nfm9kDZnaemZ0H/Ba4v3BlSUmbdT7UJIqmFTGmvoYx9dUaMCdlJ9dO6suB64GmcLne3TVATvZNTX3w7Oplv4aWF6OuJifppEZUS/nJ+TKRu9/t7peFyy8LWZSUgSP/DiqHBqOri0A62cDLLVvZ2qqOaikfvT0PYouZbc6ybDEz9djJvhs2Eg4/DxbeARtfjbqaXjWlErjDErUipIzsNSDcvc7d67Msde5eP1BFSok6+hKwGDz5vagr6dWM8BnVuswk5UR3Ikl0Ekk45Ex47qewdXDPGt9YV824RI0CQsqKAkKiNfcL0NkGT/0g6kp6lU4mWKRbXaWMKCAkWiMPgGknw9M3wo7BPZ1FUyrBinXb2LxTExhLeVBASPSOvQzatsD8G6KuZK+6+yGWaOI+KRMKCIne2DRMOQ6e+iG0bY+6mj1K7+qoHtwtHZF8UUDI4HDsF2H7enj21qgr2aORtdUkG4Zoyg0pGwoIGRz2OwomzoEnvwsdbVFXs0fpZILFupNJyoQCQgaPuZfB5lXB4LlBKp1KsHL9djZtV0e1lD4FhAweB74fxjbBE9+Brs6oq8mqKZzZdfFqtSKk9CkgZPAwC/oi1i+HZfdFXU1WM8ZrRLWUj4IGhJnNM7MXzWy5mV2R5f3zzKzFzJ4Plwsy3uvM2D44vy0k/w7+KIycAo99c1A+lnT4sComjBiiAXNSFgoWEGYWB64FPgRMA84ys2lZdr3D3WeGy40Z23dkbD+xUHXKIBOLw9zPwxuLYPlDUVeTVVOygYW61VXKQCFbELOB5e6+wt3bgNuBkwr4eVIq0qdDfSpoRQxCM5IJXt+wg7e2D967rUTyoZABkQRez1hvDrf1dKqZLTSzu8xsQsb2GjNbYGZPmdnJ2T7AzC4K91nQ0tKSx9IlUhVVMOdSeO1JePXJqKt5h+6OavVDSKmLupP618Akd28CHgRuyXhvorvPAj4BfMfMDuh5sLtf7+6z3H1WY2PjwFQsA+PQT8HQUfDY4HssaXdHtQbMSakrZECsAjJbBKlw2y7uvt7dW8PVG4HDM95bFf5cATwCHFrAWmWwqRoKR/0dLH8Q1iyMuprdJIZWMnHkUA2Yk5JXyICYD0wxs8lmVgWcCex2N5KZjctYPRFYFm4fbmbV4etRwBxgaQFrlcHoiAuguh4eH3ytiHQyoRaElLyCBYS7dwCXAA8QfPHf6e5LzOwqM+u+K+lSM1tiZi8AlwLnhdsPBhaE2x8Gvu7uCohyM6QhCIkl98K65VFXs5umVIJVb+1g/dbW3ncWKVIVhTy5u98P3N9j21cyXn8J+FKW454E0oWsTYrEUX8fPEzoiW/DSddGXc0u6WQDEHRUv+eg0RFXI1IYUXdSi+xdbSMcdg68cAdsao66ml2mJ4NHsqsfQkqZAkIGv2P+AXB48vtRV7JLfU0l+48apn4IKWkKCBn8GvaDpjPgmZth27qoq9klnUpoLISUNAWEFIc5n4eOncFT5waJdDLBmk07admijmopTQoIKQ6NU4OJ/J6+AXYOjmdCdz+CVP0QUqoUEFI8jr0MWjfBgh9HXQkA05MJzDSiWkqXAkKKx/hD4YD3w/9eC+07oq6G2uoK9h81TP0QUrIUEFJcjr0MtrXAcz+LuhIAmlINLNLU31KiFBBSXCbOgQlHwhPfhc7onwudTiZ4c3MrazfvjLoUkbxTQEhx6X4s6abXYNFdUVdDWlN/SwlTQEjxmXIcjJkBj38buroiLWXauHpi6qiWEqWAkOJjBnO/AOtehBd/G2kpw6orOHB0rVoQUpIUEFKcpp8CI/YPHkvqHmkpM5LBiGqPuA6RfFNASHGKxYPR1aufgxUPR1pKUzJBy5ZW3tysEdVSWhQQUrwOORPqxkX+WNJ0Kpj6e2GzbneV0qKAkOJVUR3M9LryMXj96cjK6O6o1pQbUmoUEFLcDjsXhoyItBUxpCrO1DF1LFRASIlRQEhxq66Fo/4OXvodvLkksjLSyQSLmtVRLaVFASHFb/aFUFUbjIuISFMqwfptbazepBHVUjoUEFL8hgyHWefD4rthw4pISpgRTv29SAPmpIQoIKQ0HH0xxCrhvksjmen14HH1VMRME/dJSVFASGmoGwsnfR9WPg53nA0dAzsmoaYy7KhWC0JKiAJCSkfT6XDid2H5H+F/Pj3gs72mkwkWa0S1lBAFhJSWw86BE74RzNF09wXQ2TFgH51OJdi4vZ3mjdE/zEgkHyqiLkAk72ZfGFxi+sOXg8F0J/8wmJqjwJoypv6eMGJowT9PpNDUgpDSdMwl8L5/gYV3wG8+PyDTgh80to7KuGlmVykZakFI6Xr35UFL4tH/gooa+NB/BlOFF0h1RZyDxtbpVlcpGQoIKW3v/XJw2+v/fj+43PTBqwsaEulkA/cvWoO7YwX8HJGBoEtMUtrM4Lh/hyMuhCe/Bw//R0E/Lp1MsGlHO69vUEe1FD+1IKT0mQWXlzpb4dH/DFoS7/7HgnxUd0f1wlVvsd9IdVRLcVMLQspDLAYf+Q40nQF/uhqe/H5BPmbqmDqq4jH1Q0hJUAtCykcsDif9YPdbYGdfmNePqKqI8a5xdbqTSUqCAkLKS7wCTr0ROtvg/n8MQuKwc/L6EelkgvteWE1XlxOLqaNaipcuMUn5iVfCx2+GAz8QTO638M68nr4plWDLzg5e3bA9r+cVGWgKCClPFdVwxs9g0lz45Wdgyb15O3X31N96RrUUOwWElK/KIfCJOyA1G+7+W3jxd3k57dQxdVRVxPSMail6Cggpb1XD4JP/A+MOgTvPCWaC7afKeIxp4+o19bcUvYIGhJnNM7MXzWy5mV2R5f3zzKzFzJ4Plwsy3jvXzP4aLucWsk4pczX1cPbd0HgQ3P5JeOWxfp8ynUywZPVmuro09bcUr4IFhJnFgWuBDwHTgLPMbFqWXe9w95nhcmN47AjgSuBIYDZwpZkNL1StIgwZDp+6F4ZPhl+cAa891a/TpVMJtrZ28Mr6bXkqUGTgFbIFMRtY7u4r3L0NuB04KcdjjwcedPcN7r4ReBCYV6A6RQLDRsE5v4L6cfCz02DVM/t8ql1Tf+sykxSxQgZEEng9Y7053NbTqWa20MzuMrMJfTnWzC4yswVmtqClpSVfdUs5qxsD59wHQ0fATz8Gaxbu02kObKylpjKmfggpalF3Uv8amOTuTQSthFv6crC7X+/us9x9VmNjY0EKlDKUSMK5v4aqWvjpybB2WZ9PURF2VOtOJilmhQyIVcCEjPVUuG0Xd1/v7t1Pl78RODzXY0UKavhEOPc+iFXCrSfBuuV9PkVTqoHFqzfRqY5qKVKFDIj5wBQzm2xmVcCZwH2ZO5jZuIzVE4HuX9UeAI4zs+Fh5/Rx4TaRgTPygCAkujrh1hNh48o+HZ5OJtje1smKlq2FqU+kwAoWEO7eAVxC8MW+DLjT3ZeY2VVmdmK426VmtsTMXgAuBc4Lj90AXE0QMvOBq8JtIgOr8aCg47p9O9zyUdjUnPOh6YxnVIsUI3MvjebvrFmzfMGCBVGXIaVq9XNwy4nBnU6f/h3Uje31kM4uZ8aVD3DGERP46onTB6BIkb4zs2fcfVa296LupBYpDuMPDQbTbV0bBMXW3u+ai8eMGcl6tSCkaCkgRHI1YTZ84k5467Xg7qbtvV/1nJFMsHT1Zjo6uwagQJH8UkCI9MWkOXDWbbDur/DTU2Dn3lsHTakEO9o7eblFI6ql+CggRPrqgPfCGT+FN5cEI65bt+xx13SyAdDU31KcFBAi+2Lq8XDaTcF0HL84E9qyPxxo/1HDqKuu4KpfL+X8m+fzg0eWs2DlBlo7Oge4YJG+011MIv2x6C64+4KgVXHmbVBZ845dHv/rOn67aDVPv7Jh16WmqooYM1MNHDF5OEdMGsFhE4dTX1M50NWL7PUuJgWESH8993P41d/D1Hlw+k+homqPu67f2sqCVzcy/5UNzH91I4tXBSOtYwbvGlvP7MkjOGLSCI6YNJzR9e8MG5F8U18itccAAAnHSURBVECIFNqCm+A3X4CDT4TTfgLxipwO297WwXOvvcXTr2xgwasbePbVt9jRHlx+mjhyKEdMGsHsSSOYNWk4k0cNw8wK+aeQMqSAEBkIT/0Qfn8FpD8Op1wHsXifT9He2cWS1ZtZsHJDGBob2bCtDYBRtdUcMWk4s8LQOHhcHRVxdSNK/+wtIHL7NUdEenfU30HHTvjjVyFeDSd+D2J9+wKvjMeYOaGBmRMauODY/XF3Xm7ZxvyVG8LLUhv43eI3ABhWFeewicPDS1IjOHS/Bmoq+x5KInuigBDJp7lfgPad8OevQ0U1fPib0I/LQmbGgaNrOXB0LWfN3g+ANZt2MH/lxl2tjG//8SXcoTJuzEgmmB0GxqxJw2kYuuf+EJHe6BKTSL65B62IJ74DR10Mx1/Tr5Dozabt7Tzz2gbmrww6vxc2b6ItHLk9dUztrhbGEZNHkGwYkp8P7eqCzlboaIXOtqDl1BH+7GyFjjY623bQ2rqD9tYdtLXuoKNtB+2tO+lsa6WzfQedba10te+gq6MV72iF9lbobMU6WsE7g79HwsUBHMvcBrv2sYx98HA9fB3uiPXcPzzGcBzCc7z9nrH7Z3bv15uc/6Vz/O61HD51fe0UDr/snlw/effz6xKTyAAygw98NfjyfOraoCXx/q/sPSS6uqCrHTrbw58dGesdGdt7rneQ6GrnfV3tvC/VDuM6aG9vpXn9Zl5r2Uzzus288fwWXlrQxit0MrzGSCUqSdZVMLYuzpBYF53tO+hqb6WrfSfeHn5Zd+yEzjassxXrbCPe2Uqsq414Vxtxb6PCO3r9a4gDQ3vZp83jtFFJK5W0UUk7FXRYJZ1UgHV/TdvbX9dmu7Z1c+wd290yj7Nd/y671rtfd3+GGcbbr3c71nh7Peeg7+d+PTb7bm+985i2Ydke1tl/CgiRQjCDeV8Lvmgf/xYsuQe8a89f/J6/uZoqgcnhEtQSbgS6OmO0b4jTvj5OB3E2U0kbFbR6ZfhFXUErVbR5BW1U00pt+H5V8MUdr8JjVXhFNV0V1RCvwipqoKKaeGU1VlFDvKqGWFUNFZU1VFbVEK8eQmVVDVXVQ6iqGUJ1zVCqa4ZQUzOEIdWVDKmMM6QqzvCKOLGY7tIaTBQQIoViBh/+FtQnYe1SiFcGT6iLV4Q/KyFWsYftPdZ3bevtHHs/ZywWo8qdNzZsZ8HKjWxt7WBIVTz4kq6MM7QqTk1VnETGtiFVcWoq41Tqjqmyo4AQKaRYDP7m8qir2I2ZMXHkMCaOHBZ1KTLI6VcCERHJSgEhIiJZKSBERCQrBYSIiGSlgBARkawUECIikpUCQkREslJAiIhIViUzWZ+ZtQCv9uMUo4B1eSonn1RX36iuvlFdfVOKdU1098Zsb5RMQPSXmS3Y04yGUVJdfaO6+kZ19U251aVLTCIikpUCQkREslJAvO36qAvYA9XVN6qrb1RX35RVXeqDEBGRrNSCEBGRrBQQITP7qpmtMrPnw+WEqGvqycy+aGZuZqOirgXAzK42s4Xh39cfzGx81DUBmNl/mdn/hbX90swaoq4JwMw+bmZLzKzLzCK9E8bM5pnZi2a23MyuiLKWTGZ2k5mtNbPFUdeSycwmmNnDZrY0/Df8XNQ1AZhZjZk9bWYvhHX9Wz7Pr4DY3bfdfWa43B91MZnMbAJwHPBa1LVk+C93b3L3mcBvgK9EXVDoQWCGuzcBLwFfiriebouBjwGPRlmEmcWBa4EPAdOAs8xsWpQ1ZbgZmBd1EVl0AF9092nAUcDFg+TvrBV4n7sfAswE5pnZUfk6uQKieHwb+Cd2f355pNx9c8bqMAZJbe7+B3fvCFefAlJR1tPN3Ze5+4tR1wHMBpa7+wp3bwNuB06KuCYA3P1RYEPUdfTk7mvc/dnw9RZgGZCMtirwwNZwtTJc8vb/oQJid5eElyVuMrPhURfTzcxOAla5+wtR19KTmV1jZq8Dn2TwtCAynQ/8LuoiBpkk8HrGejOD4MuuWJjZJOBQ4C/RVhIws7iZPQ+sBR5097zVVVbPpDazPwJjs7z1ZeCHwNUE6Xs18E2CL5fBUNs/E1xeGnB7q8vdf+XuXwa+bGZfAi4BrhwMdYX7fJng0sDPB6KmXOuS4mVmtcDdwOd7tKAj4+6dwMywr+2XZjbD3fPSh1NWAeHuH8hlPzO7geCa+oDZU21mlgYmAy+YGQSXS541s9nu/kZUdWXxc+B+BiggeqvLzM4DPgK83wfwXu4+/H1FaRUwIWM9FW6TvTCzSoJw+Lm73xN1PT25+1tm9jBBH05eAkKXmEJmNi5j9RTy9BfcX+6+yN1Hu/skd59EcDngsIEIh96Y2ZSM1ZOA/4uqlkxmNo+gv+ZEd98edT2D0HxgiplNNrMq4EzgvohrGtQs+O3sx8Ayd/9W1PV0M7PG7rv0zGwI8EHy+P+hBsqFzOynBHcBOLAS+Iy7r4m0qCzMbCUwy90jn1HSzO4GDgK6CGbS/ay7R/6bqJktB6qB9eGmp9z9sxGWBICZnQJ8D2gE3gKed/fjI6rlBOA7QBy4yd2viaKOnszsNuA9BLOTvglc6e4/jrQowMzmAo8Biwj+ewf456jvdjSzJuAWgn/HGHCnu1+Vt/MrIEREJBtdYhIRkawUECIikpUCQkREslJAiIhIVgoIERHJSgEh0gdmtrX3vfZ6/F1mtn/4utbMrjOzl83sGTN7xMyONLMqM3vUzMpqIKsMPgoIkQFiZtOBuLuvCDfdSDAx3RR3Pxz4NDAqnEDvIeCMaCoVCSggRPaBBf7LzBab2SIzOyPcHjOzH4TPo3jQzO43s9PCwz4JdM8TdQBwJPAv7t4F4O6vuPtvw33vDfcXiYyasCL75mMEI+8PIRj1O9/MHgXmAJMInrMwmmBa6JvCY+YAt4WvpxOMpO7cw/kXA0cUpHKRHKkFIbJv5gK3uXunu78J/JngC30u8D/u3hXOl/VwxjHjgJZcTh4GR5uZ1eW5bpGcKSBEBs4OoCZ8vQQ4JHy6255UAzsLXpXIHiggRPbNY8AZ4cNaGoF3A08DTwCnhn0RYwgmnuu2DDgQwN1fBhYA/xbOFIqZTTKzD4evRwLr3L19oP5AIj0pIET2zS+BhcALwJ+AfwovKd1NMCX7UuBnwLPApvCY37J7YFwAjAGWm9ligucxrw3fe2+4v0hkNJurSJ6ZWa27bw1bAU8Dc9z9jXC+/ofD9T11Tnef4x7gCnd/aQBKFslKdzGJ5N9vwoe4VAFXdz/cyd13mNmVBM9/fm1PB4cP8blX4SBRUwtCRESyUh+EiIhkpYAQEZGsFBAiIpKVAkJERLJSQIiISFYKCBERyer/AxrJKZha5fwaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 对不同Cs将所有交叉验证集的结果求平均\n",
    "scores_L1 = -np.mean(lrcv_L1.scores_[1], axis = 0) \n",
    "scores_L2 = -np.mean(lrcv_L2.scores_[1], axis = 0)\n",
    "# 画出不同Cs下的模型结果曲线图\n",
    "plt.figure()\n",
    "plt.plot(np.log10(Cs), scores_L1, label='l1') \n",
    "plt.plot(np.log10(Cs), scores_L2, label='l2') \n",
    "plt.legend()\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('logloss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params for LogisticRegressionCV L1: 1\n",
      "Best params for LogisticRegressionCV L2: 0\n",
      "Best score for LogisticRegressionCV L1: 0.479106\n",
      "Best score for LogisticRegressionCV L2: 0.478870\n"
     ]
    }
   ],
   "source": [
    "# 得到模型最好结果对应的C参数\n",
    "best_C_L1 = np.argmin(scores_L1)\n",
    "best_C_L2 = np.argmin(scores_L2)\n",
    "print(\"Best params for LogisticRegressionCV L1: %d\" % Cs[best_C_L1]) \n",
    "print(\"Best params for LogisticRegressionCV L2: %d\" % Cs[best_C_L2])\n",
    "# 得到模型的最好结果\n",
    "best_score_L1 = np.min(scores_L1)\n",
    "best_score_L2 = np.min(scores_L2)\n",
    "print(\"Best score for LogisticRegressionCV L1: %f\" % best_score_L1) \n",
    "print(\"Best score for LogisticRegressionCV L2: %f\" % best_score_L2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score=nan,\n",
       "             estimator=LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                          fit_intercept=True,\n",
       "                                          intercept_scaling=1, l1_ratio=None,\n",
       "                                          max_iter=100, multi_class='auto',\n",
       "                                          n_jobs=None, penalty='l2',\n",
       "                                          random_state=None, solver='lbfgs',\n",
       "                                          tol=0.0001, verbose=0,\n",
       "                                          warm_start=False),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'C': [1e-05, 0.0001, 0.001, 0.01, 0.1, 1, 10, 100,\n",
       "                               1000]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "             scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "# 设置正则项系数搜索空间\n",
    "Cs = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 10, 100, 1000] \n",
    "tuned_parameters = dict(C=Cs)\n",
    "# 创建L1和L2正则的logistic回归估计器\n",
    "lr_L1 = LogisticRegression(penalty='l1', solver='liblinear') \n",
    "lr_L2 = LogisticRegression(penalty='l2', solver='lbfgs')\n",
    "# 创建L1和L2正则的GridSearchCV网格搜索器\n",
    "grid_L1 = GridSearchCV(lr_L1, tuned_parameters, cv=3, scoring='neg_log_loss', return_train_score=True)\n",
    "grid_L2 = GridSearchCV(lr_L2, tuned_parameters, cv=3, scoring='neg_log_loss', return_train_score=True)\n",
    "# 训练估计器 \n",
    "grid_L1.fit(X_train, y_train) \n",
    "grid_L2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score for gridsearch L1:  0.4791118456911286\n",
      "Best params for gridsearch L1:  {'C': 1}\n",
      "\n",
      "Best score for gridsearch L2:  0.47886954117570046\n",
      "Best params for gridsearch L2:  {'C': 0.1}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best score for gridsearch L1: \", -grid_L1.best_score_)\n",
    "print(\"Best params for gridsearch L1: \", grid_L1.best_params_)\n",
    "print(\"\\nBest score for gridsearch L2: \", -grid_L2.best_score_)\n",
    "print(\"Best params for gridsearch L2: \", grid_L2.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXxcdb3/8dcnk61rmklToE3bTGtBylZomoAsgkipoKCCWBTZRORecLkq/kC8cAFRvF6v3uvFBVlERRZFtEIVkF20SwoF2rJY2gJpCy1J9zbN9vn9cU7aaZi2k3YmJzPzfj4ex8w58z1nPq103jnn+z3fY+6OiIhIT0VRFyAiIv2TAkJERFJSQIiISEoKCBERSUkBISIiKRVHXUCmDB8+3Gtra6MuQ0Qkp8ybN+8dd69O9V7eBERtbS2NjY1RlyEiklPM7PWdvZfVS0xmNs3MXjGzxWZ2RYr3f2Bm88PlVTNbm/TeeWb2z3A5L5t1iojIu2XtDMLMYsBNwElAEzDXzGa4+6LuNu7+b0ntvwAcHr6OA9cAdYAD88J912SrXhER2VE2zyDqgcXuvsTd24C7gdN30f5s4K7w9cnAI+7eEobCI8C0LNYqIiI9ZLMPYhTwZtJ6E9CQqqGZjQUSwGO72HdUiv0uBi4GGDNmzN5XLCKSpvb2dpqammhtbY26lLSUl5dTU1NDSUlJ2vv0l07q6cDv3L2zNzu5+83AzQB1dXWaVEpE+kxTUxNDhgyhtrYWM4u6nF1yd5qbm2lqaiKRSKS9XzYvMS0HRiet14TbUpnO9stLvd1XRKTPtba2UlVV1e/DAcDMqKqq6vXZTjYDYi4wwcwSZlZKEAIzejYys/cClcA/kjY/BEw1s0ozqwSmhttERPqNXAiHbntSa9YuMbl7h5ldRvDFHgNuc/eFZnYd0Oju3WExHbjbk+Ydd/cWM7ueIGQArnP3lmzUubmtg58+8Vo2Di19zYyPThrJuOrBUVciktInfxb8HnzP54+KuJL0ZLUPwt1nAjN7bLu6x/p/7GTf24DbslZcaEtbJz96fHG2P0b6gDu80byJH04/POpSRPrE4MGD2bhxIwDTpk1j1qxZHHPMMTzwwAMZOX5/6aSOTNXgMpZ+59Soy5AMuPTOZ5m9tAV3z6lTf5FMuPzyy9m8eTM/+9nPMnZMTdYneaM+EWflulaa1myJuhSRPnfiiScyZMiQjB6z4M8gJH/UJ+IAzFnawuj4wIirkUJy7Z8WsmjF+t22W7QyaNPdF7ErE0cO5ZqPHLTXte0NnUFI3jhgnyFUDChhztKsjGcQKTg6g5C8UVRkTKmNM3tpc9SlSIFJ9zf9XBvFpDMIySsNiTjLmjfz9vrcmP5ApD9TQEheSe6HECkkxx57LJ/4xCd49NFHqamp4aGH9v7eYl1ikrxy0MihDCqNMWdpCx85bGTU5YhkVfc9EABPP/10xo+vgJC8UhwrYnJtXGcQ0i/lSt9DN11ikrzTkIjzytsbWLOpLepSRHKaAkLyzrZ+iGU6ixDZGwoIyTuH1lRQWlyky0wie0kBIXmnrDjG4aOHKSBE9pICQvJSQyLOwhXr2NDaHnUpItvdfmqw5AgFhOSlhnFVdDnMe31N1KWIZM3gwcGzT+bPn89RRx3FQQcdxKGHHso999yTkeNrmKvkpcPHDKO4yJi9tIXjDxgRdTkiWTVw4EB++ctfMmHCBFasWMHkyZM5+eSTGTZs2F4dVwEheWlgaTGH1FSoH0IKwv7777/t9ciRIxkxYgSrV69WQIjsTH0izm1/W8qWtk4GlMaiLkfy2Z+vgLde3H27t14IfqbTD7HvIfChG3tdypw5c2hra2P8+PG93rcn9UFI3mpIxGnvdJ57U/0QUhhWrlzJZz7zGW6//XaKivb+611nEJK36mrjmAUT971v/PCoy5F8lu5v+t1nDhc8mPES1q9fz6mnnsoNN9zAkUcemZFj6gxC8tbQ8hIm7jeU2UvUDyH5ra2tjY997GOce+65nHnmmRk7rgJC8lp9Is6zb6yhraMr6lJEsubee+/lqaee4he/+AWTJk1i0qRJzJ8/f6+Pq0tMktcaEnFuf2YZLy5fy+Sx8ajLEcmo7um+zznnHM4555yMH18BIXltSm0QCrOXtiggJHpZ6HvIJl1ikrxWNbiMCSMG634IkT2ggJC8V5+I07hsDZ1dHnUpkmfcc+e/qT2pVQEhea8+EWfj1g4WrVgfdSmSR8rLy2lubs6JkHB3mpubKS8v79V+6oOQvNf9AKHZS5s5pKYi4mokX9TU1NDU1MTq1aujLiUt5eXl1NTU9GofBYTkvf0qBjAmPpA5S1u46NhxUZcjeaKkpIREIhF1GVmV1UtMZjbNzF4xs8VmdsVO2pxlZovMbKGZ/SZpe6eZzQ+XGdmsU/JfQyLO3GUtdKkfQiRtWTuDMLMYcBNwEtAEzDWzGe6+KKnNBOBK4Gh3X2NmyfMyb3H3SdmqTwpLfSLOb+c1sXj1RvbfZ0jU5YjkhGyeQdQDi919ibu3AXcDp/do8zngJndfA+Duq7JYjxSwhkQVALOXNEdciUjuyGZAjALeTFpvCrcl2x/Y38yeMbNZZjYt6b1yM2sMt3801QeY2cVhm8Zc6SiSaIyOD2DfoeXM1v0QImmLupO6GJgAHA/UAE+Z2SHuvhYY6+7LzWwc8JiZvejuryXv7O43AzcD1NXV6eKy7JSZUZ+IM2tJMCzRzKIuSaTfy+YZxHJgdNJ6TbgtWRMww93b3X0p8CpBYODuy8OfS4AngMOzWKsUgIZxcVZt2MrrzZujLkUkJ2QzIOYCE8wsYWalwHSg52ikPxCcPWBmwwkuOS0xs0ozK0vafjSwCJG90BDeD6FpN0TSk7WAcPcO4DLgIeAl4F53X2hm15nZaWGzh4BmM1sEPA5c7u7NwIFAo5k9H26/MXn0k8ieGF89mPigUmYtVUe1SDqy2gfh7jOBmT22XZ302oGvhEtym78Dh2Sztm3aW+F/JkGsGI66DAbGw6UKBoQ/SweBrlnnPDOjvjauMwiRNEXdSR291nWweTV0dcBf/l/qNrGypNCoDH4OrNq+bVuYJIVL6WCFSj9Un4jzl4VvsXztFkYNGxB1OSL9mgJiyD5wdTN0dYZh0QKbm4NlS9LrzS3BsqUFVi0K318DvpMnlcVKt5+BpDorSd7eva1siEIlyxrGBf0Qc5e2MOrwnqOuRSSZAqJbUWz7FzbvSW+fri5oXbs9OHYIk+btIbK5GVa9tL3dzkKlqGTHIFn9MgwaAZf+I2N/zEL33n2HMqS8mNlLW/ioAkJklxQQe6OoKClU0tQdKt3B8a5Aadl+trJ1PWx+B+76FHzoRhg2Jnt/lgIRKzKm1MaZrY5qkd1SQPS15FCpGr/rth1tMOvH8OR34f/q4f1fDzrSi0v7ptY8VZ+I89jLq1i9YSvVQ8qiLkek39IDg/qz4lI45stw6Rx4z4nw6LXw02Ng6dNRV5bTup8PMXeZRjOJ7IoCIhcMGw3T74Sz74GOLXDHh+H3F8NGzW24Jw4ZVcGAkpiGu4rshgIilxwwDf51Nhz7NVjwe/hRHcz5eTACS9JWEiti8thKTdwnshsKiFxTOhBO/Hf413/AyMNg5tfglhNh+bNRV5ZT6hNxXn5rPes2t0ddiki/pYDIVcMnwLkz4IxbYf0K+PkH4MGvwpa1UVeWE+oTcdzVDyGyKwqIXGYGh5wJl82F+ouh8Tb4vzp4/h5wzX6+K5NGD6M0VsQcBYTITikg8kF5BZzyn/C5x4N7Je6/GO74CKx+JerK+q3ykhiHja5QP4TILigg8snISfDZR+DDP4C3XoCfHA1//Q9o0/MPUmlIVLFg+To2be2IuhSRfkkBkW+KYlB3IVw2Dw49C/72A7ipAV6euft9C0x9Ik5nl/PsG2uiLkWkX1JA5KvB1fDRH8MFfw6mK7/7bLjrbFjzetSV9RtHjK0kVmTMXqLLTCKpKCDy3dj3wSVPw0nXw5Ing7OJp78fTONR4AaXFXPwyKG6YU5kJxQQhSBWAkd/ES6bAxM+CI9eBz89GpY+FXVlkatPxJn/5lpa23WzoUhPCohCUlEDn/w1fOq30LE1GOl03+dgw9tRVxaZhkQVbZ1dPP+m7h8R6UkBUYj2nwqXzobjvg6L/gD/N6Vgp+yYUhvHDF1mEklBAVGoSgbAB66Cf/l7MDx25teCu7GXz4u6sj5VMbCEA/YZovshRFJQQBS64RPg3D8GU3ZsWAk/PxEe+ErwQKMC0ZCIM+/1NbR37uRJfyIFSgEhO07Z0XAJzLs9uOz0/N0FMWVHfaKKLe2dLFi+LupSRPoVBYRsV14RPNr04iehshbu/zz84sOw6uWoK8uq7gcIqR9CZEcKCHm3/Q6FCx+Gj/wPvL0gGBL7yDXQtinqyrKiekgZ46oHKSBEelBASGpFRTD5fPjCPDh0Ojzzw3DKjgejriwrGhJx5ixrobMr/y+piaRLASG7Nmg4fPQmuOAvUDYE7v4UfLcWfn5S1JVlVH0izobWDl5+a33UpYj0GwoISc/Yo+DzT8HUb0HrOmj+Z9QVZVR9ogpQP4RIMgWEpC9WAu/7AgwbC61r4NWHo64oY0YNG0BN5QAFhEgSBYT03qVzID4eHvoGdObPM53rE3HmLG3BC2Bor0g6shoQZjbNzF4xs8VmdsVO2pxlZovMbKGZ/SZp+3lm9s9wOS+bdUovFZfCyTcEl5nm3hp1NRnTkIjTvKmN11bn52gtkd7KWkCYWQy4CfgQMBE428wm9mgzAbgSONrdDwK+HG6PA9cADUA9cI2ZVWarVtkD+0+DcSfAE9+BzflxWaa7H2L20uaIKxHpH7J5BlEPLHb3Je7eBtwNnN6jzeeAm9x9DYC7rwq3nww84u4t4XuPANOyWKv0lhmc/G3Yuj4IiTxQWzWQ6iFl6ocQCWUzIEYBbyatN4Xbku0P7G9mz5jZLDOb1ot9JWr7TAwebzr3Vlj1UtTV7DUzoz4RZ/YS9UOIQPSd1MXABOB44Gzg52Y2LN2dzexiM2s0s8bVq1dnqUTZpeO/AWWDgw7rPPhSPTIR5631rTSt2RJ1KSKRy2ZALAdGJ63XhNuSNQEz3L3d3ZcCrxIERjr74u43u3udu9dVV1dntHhJ06AqeP8V8Npj8M/cH/a6vR9Cl5lEshkQc4EJZpYws1JgOjCjR5s/EJw9YGbDCS45LQEeAqaaWWXYOT013Cb90ZSLoOo9eTHsdcKIwQwbWMLsJeqoFslaQLh7B3AZwRf7S8C97r7QzK4zs9PCZg8BzWa2CHgcuNzdm929BbieIGTmAteF26Q/Ki6FqTdA8+LgyXQ5rKjImFIbzMskUugsXzrj6urqvLGxMeoyCpc7/PrjwRPpvvBccOkpR93y9BK+9eBLzLryRPatKI+6HJGsMrN57l6X6r2oO6klX2wb9roRnvh21NXslYbueZl0FiEFTgEhmTPiwGDYa+Nt8PaiqKvZYwfuN4TBZcXM0Q1zUuAUEJJZx18ZTAuew8Nei2NFTB5bqRvmpOApICSzBlUFIbHkcXg1dwee1SfivPr2Rlo2tUVdikhkFBCSeVMugqoJ8PBV0JGbX7ANek61iAJCsiBWEnRYNy+Gubk57PXQmmGUFRcpIKSgKSAkOyacBONPhCe+C5tyr7O3tLiII8ZUMmdZ7tUukim9DggzKzKzodkoRvJI97DXto3w+A1RV7NH6hNxFq1Yz/rW3L47XGRPpRUQZvYbMxtqZoOABcAiM7s8u6VJzhvxXpjyWZh3e04Oe21IxOlymLdsTdSliEQi3TOIie6+Hvgo8GcgAXwma1VJ/jj+SigbCg9dmXPDXg8fU0lJzDRxnxSsdAOixMxKCAJihru3A7n1r12iMTAeDnt9Al79S9TV9MqA0hiH1gzTDXNSsNINiJ8By4BBwFNmNhZYn62iJM9M+SwM3x8eyr1hr/WJOC80rWNLW2fUpYj0ubQCwt3/191HufspHngdOCHLtUm+6B722vIazLk56mp6pT4Rp6PLee4N9UNI4Um3k/pLYSe1mdmtZvYs8IEs1yb5ZMJJ8J4PwpP/CZveibqatE0eW0mRwSz1Q0gBSvcS04VhJ/VUoJKgg/rGrFUl+SkHh70OLS9h4sih6oeQgpRuQFj48xTgV+6+MGmbSHqqDwim4Zj3C3hrQdTVpK0hUcVzb6xla4f6IaSwpBsQ88zsYYKAeMjMhgBd2StL8tbxV4TDXnNnttf6RJytHV282LQu6lJE+lS6AfFZ4ApgirtvBkqBC7JWleSvgXE44Ruw9El45c9RV5OWKbXBxH26H0IKTbqjmLqAGuCbZvZfwPvc/YWsVib5q+5CGH5AONvr1qir2a34oFL232ewAkIKTrqjmG4EvgQsCpcvmlluP1dSorNt2OuSnBn2Wp+IM29ZCx2durIqhSPdS0ynACe5+23ufhswDfhw9sqSvDfhgzBhajDsdePqqKvZrfpEFZvaOlm0UveHSuHozWyuw5JeV2S6EClAU2+A9s05MexVDxCSQpRuQHwHeM7MfmFmdwDzgP7/r1r6t+r9Ycrn4Nk7+v2w132GllNbNVD9EFJQ0u2kvgs4Evg9cB9wlLvfk83CpEC8/+tQXgF/uaLfD3utT8SZu6yFrq7+XadIpuwyIMzsiO4F2A9oCpeR4TaRvTMwDidcBcuehpcfjLqaXapPVLF2czuvrtoQdSkifaJ4N+9/fxfvOZqPSTJh8gUw9xZ4+JvBnE3FZVFXlFJyP8R799VDFSX/7fIMwt1P2MWicJDMiBUHw17XLIXZP426mp2qqRzAyIpy9UNIwdjdGQQAZvbxFJvXAS+6+6rMliQF6T0nwoST4cnvwWFnw+ARUVf0LmZGfSLOM6814+6YaToyyW+9mWrjFuDT4fJz4P8Bz5iZHj0qmXHyDdCxBR77VtSV7FR9oorVG7ayrHlz1KWIZF26AVEMHOjuZ7j7GcBEgj6IBoKgENl7wydA/cXw7C9hZf+cyaU+7IeYvUTTf0v+SzcgRrv720nrq8JtLUD7znYys2lm9oqZLTazK1K8f76ZrTaz+eFyUdJ7nUnbZ6T7B5Ic9/6vw4DKfjvb6/jqQQwfXKob5qQgpNUHATxhZg8Avw3Xzwy3DQLWptrBzGLATcBJBENj55rZDHdf1KPpPe5+WYpDbHH3SWnWJ/liQGUw2+vMr8HLD8CBH4m6oh1090Ooo1oKQbpnEJcCtwOTwuUO4FJ33+TuO3s2dT2w2N2XuHsbcDdw+t4WLAVg8gVQfWAw7LUfzvZaXxtn+dotNK1RP4Tkt3TvpHbgb8BjwKPAU+G2XRkFvJm03hRu6+kMM3vBzH5nZqOTtpebWaOZzTKzj6b6ADO7OGzTuHp1/5/wTdIUK4Zp34Y1y2DWT6Ku5l3qE1UAzF2mswjJb+lO930WMIfg0tJZwGwzOzMDn/8noNbdDwUeITgz6TbW3euATwE/NLPxPXd295vdvc7d66qrqzNQjvQb4z8A+0+Dp/4LNvavkdQH7DuEoeXFzF6igJD8lu4lpqsIniZ3nrufS3D56N93s89yIPmMoCbcto27N7t79zWEW4DJSe8tD38uAZ4ADk+zVskXU7uHvV4fdSU7iBUZU2rj6qiWvJduQBT1uCGuOY195wITzCxhZqXAdGCH0Uhmtl/S6mnAS+H2SjMrC18PB44meFCRFJLh74H6z8Ozv+p3w14bxsVZ8s4mVm1ojboUkaxJNyD+YmYPhcNSzwceBGbuagd37wAuAx4i+OK/190Xmtl1ZnZa2OyLZrbQzJ4HvgicH24/EGgMtz8O3Jhi9JMUgu5hr3+5sl8Ne93WD7F0TcSViGSP7b6vOWxodgbBb/IAT7v7/Vmrag/U1dV5Y2Nj1GVINsy9BR78Kpz1K5h42u7b94H2zi4Ou/ZhPjG5hmtPPzjqckT2mJnNC/t73yXd+yBw9/sIngUh0reOOB/m3hrO9joVSsqjroiSWBGTx1bqfgjJa7t7HsQGM1ufYtlgZno4r/SN7tle174Os/vPsNf62jivvL2BtZvboi5FJCt2N933EHcfmmIZ4u6aEF/6zvgT4IBT4Knvw4a3d9++DzSMq8Id5i5TP4Tkp3Q7qUWiN/Vb0NHab4a9HlpTQWlxEXOWauI+yU8KCMkdVeOh4fPw3K9h5fPvfv/2U4Olj5SXxJg0epjuh5C8pYCQ3HLc5cFzrPvJsNeGRJwFK9azcWtH1KWIZJwCQnLLgGFwwlXw+jPwUvSzwNcn4nR2OfNeVz+E5B8FhOSeI86DEQcFw17bo72T+YgxlcSKTP0QkpcUEJJ7umd7XfsGzPpxpKUMKivmkFEV6oeQvKSAkNw07ng44FR4+vuw4a1IS2lIxHn+zXW0tndGWodIpikgJHdNvT54oFDEw17rE3HaOruY/2bKhyuK5CwFhOSuqvFw5CXw3J2wYn5kZdSNjWOGng8heUcBIbntuMthYFWkw14rBpbw3n2HMmeZOqozYeG3j2Hht4+Juox3KcS6FBCS28or4APfhDf+Dpuj+4JuSMSZ9/oa2jq6IqtBJNMUEJL7jjgX9jkY1iwFj+YLuiERp7W9iwUr1kXy+SLZoICQ3FcUg2nfgc6t0LIEuvp+NNGURBwgt4a79vHUJJJ7FBCSHxLHwdBRsPEtuOcz0LapTz9++OAyxlcPYvYS9UNI/lBASP6oTEDlOHj1z3D7KX1+f0R9oorGZWvo7Ip+jqh0LFy5joUrdUlMdk4BIfll6EiYfhe880/4+Ynw9sI+++gjx8XZsLWDl1bqWVqSHxQQkn8OmAYX/hm8E249GRb/tU8+dkptDvZDiOyCAkLy036HwUWPQmUt3HkWNN6e9Y8cOWwAo+MDFBCSNxQQkr8qRgVnEu85ER74Mjz879CV3WGw9bVVzFnWgveDZ1WI7C0FhOS3siFBn8SUz8Hf/xd+ex60bc7axzUk4rRsamPxqo3btn3yZ//gkz/7R9Y+UyRbFBCS/2LFcMr3YNqN8NKf4I4Pw8ZVWfmohnFBP8RsXWaSPKCAkMJgBkf+C0y/E1a9FIxwWvVyxj9mTHwg+wwtUz+E5AUFhBSW954K5z8Y3HV961RY8kRGD29m1CeqmLN0ez/E1c2Xc3Xz5Rn9HJG+oICQwjPqiGCEU8Uo+PUZ8OyvMnr4+kSct9a38mbLloweV6SvKSAkf1zwYLCkY9houPChYIqOGZfBX6/N2AinhnBepll6TrXkOAWEFK7yofCpe2HyBfC3/4b7LoT2vf+tf8KIwcQHlaofQnJeVgPCzKaZ2StmttjMrkjx/vlmttrM5ofLRUnvnWdm/wyX87JZpxSwWAl8+Adw0vWw8A9wx2mw6Z29OqSZMaW2UgEhOS9rAWFmMeAm4EPAROBsM5uYouk97j4pXG4J940D1wANQD1wjZlVZqtWKXBmcPQX4axfwlsvwi0nwupX9+qQ9Ykq3mjZzMp16oeQ3JXNM4h6YLG7L3H3NuBu4PQ09z0ZeMTdW9x9DfAIMC1LdYoEJp4WjHBq2wS3fhCWPr3Hh2rIxedDiPSQzYAYBbyZtN4UbuvpDDN7wcx+Z2aje7OvmV1sZo1m1rh69epM1S2FrGZyMMJpyH7wq4/B/N/s0WEO3G8oQ8qKdcOc5LSoO6n/BNS6+6EEZwl39GZnd7/Z3evcva66ujorBUoBqhwbjHAa+z74w7/AYzdAL+dWihUZdeqHkByXzYBYDoxOWq8Jt23j7s3uvjVcvQWYnO6+Ilk1YBiccx8c/hl46j/h95+Djq273y9JfaKKxas2srZrYJaKFMmubAbEXGCCmSXMrBSYDsxIbmBm+yWtnga8FL5+CJhqZpVh5/TUcJtI34mVwGk/ghOvgRd/C788HTalf29DfdgPsbCzJlsVimRV1gLC3TuAywi+2F8C7nX3hWZ2nZmdFjb7opktNLPngS8C54f7tgDXE4TMXOC6cJtI3zKDY78CZ94Oy58NOq+bX0tr10NGVVBeUsTCzjFZLlIkO4qzeXB3nwnM7LHt6qTXVwJX7mTf24DbslmfSNoO/jhU1MBd04NhsNN/E/RR7EJpcRFHjKlkwdLRu2wn0l9F3UktkjtG1wcjnAZVB5ebXrh3t7vUJ+Is6dqHjV7WBwWKZJYCQqQ34gn47MMwuiHouH7iu7sc4dSQqMIxFnWoH0JyjwJCpLcGVMI5v4fDPgVPfBvuv2SnI5wOHzOMYjq5ufUkvnH/i/xm9hu82LSOrR2dfVy0SO9ltQ9CJG8Vl8JHfwzxcfD4t2BdE3zyVzAwvkOz8pIYny9/hL+1v5cHnl/Bb2a/AUBJzNh/nyEcPLKCg0cN5eBRFRy431DKS2JR/GlEUlJAiOwpM3j/5VBZC3/8V7j1JPj0b4PQSHJK6XOcUvocE698mjdbtrBgxTpeXL6OBcvX8fCit7inMZg0IFZkTBgxmINGVnBIGBoTRw5lYKn+mUo09F+eyN469BPBCKe7PwW3fBCm3wVjGt7VzMwYUzWQMVUDOeWQ4BYgd2fFulZebFrHwjA4nnx1Ffc92xTuA+OrB3PIqAoOGjmUQ8LQGFJe0qd/RClMCgiRTBh7FFz0V7jzE3DHR+BjP4GDz9jtbmbGqGEDGDVsANMO3hcIQuPt9VtZsDwIjIUr1vH3197h/ue2TyaQGD6Ig0dVcHAYGgeNrKBioEJDMksBIZIpVeODkLj70/C7C6FlKRz71V4fxszYt6KcfSvK+eDEfbZtX7WhlYUr1rOgaR0LVqzj2dfX8KfnV2x7f0x84Lb+jKBvo4L4oNKM/NGkMCkgRDJpYBzO/QP88TJ47HpoWYp5F257P2BwxJByRhxQzgkHjNi2rWVTGwuWB4GxYPk6Fixfz8wX39r2/qhhA7Zdmjo4XKqHpHFPhjt0tkPn1vBnW7iErzu2b/fONtrbttLe1kp721Y62lrpaNtKR/tWOtvb6OzYSlf40zu24u1tdHUfryM8Zlcb1tmGdbVD63rM4NXvHI3hQDiM2IP/sW3rSa/Dn+a+rY0DlqKd7aR9N8O3DV3u3m44lV0dAKy8dsKu/+qSXhu9m+SR7pp3dsAU4t5Jk2dnslIFhEimFZfBx28OOkxdfFQAAAodSURBVKufvJExNojVsX1g2d/CL8NO6OrYzdIZtk1a3/Z6+zHiXR0c19XBcZ0dUN4BiQ7aa9pZv3kLGze3srl1K1te30r74jZidLGcTppjXQwuhuq2tRTTwYbvHEBRVxtFXR3EvJ0ib6fYO9L+4xpQGi67s9WLaaeYNoKf7RTTQTEdVkKHFdNppXRQTJcb7Z1BqLoZ2782bfu6hesQdNZ0v4/tsI6BJ4/o33Y827buScfbvt+Or9vWBsFbOnRk2n83sD2S9s7Oj9G2polWK8/AZ7ybAkIkG8zghCshnmDg/ZeQ6FgCvzh1749bVAJFxcFEgkWx4HWPpaSomKqiYqrKYjCgGGKD6GAIm9phfZuzfquztNXZ0FUdfFFvCr+orZSuWDFdVgrFJXisBIpKobgUj5VSFCvFioOlqLiUouIyikpKiZWUEysppbikjOKScopLSykpLae0rJziknLKysooKSunvLSU8tJiyoqLKCuJMaikiNJYEWY7fvkt/PYxABz0jT1/YFM2bKvr33Z/B31fCupqzcqxFRAi2XTYdF574PuUeBu1n/5R+CWe4ss99u4v+qBNSdL6nl+mKgYqwqXbvBtOoM1h0tcfoLS4iFhRJn7TlXyigBDJsjYro83KIHFc1KXsoNzaKbd2BpTq5jxJTVNtiIhISgoIERFJSQEhIiIpKSBERCQlBYSIiKSkgBARkZQUECIikpICQkREUlJAiIhISrqTWiTLDtqvYveNRPohnUGIiEhKCggREUlJASEiIikpIEREJCUFhIiIpKSAEBGRlBQQIiKSUlYDwsymmdkrZrbYzK7YRbszzMzNrC5crzWzLWY2P1x+ms06RUTk3bJ2o5yZxYCbgJOAJmCumc1w90U92g0BvgTM7nGI19x9UrbqExGRXcvmGUQ9sNjdl7h7G3A3cHqKdtcD3wVas1iLiIj0UjYDYhTwZtJ6U7htGzM7Ahjt7g+m2D9hZs+Z2ZNmdmyqDzCzi82s0cwaV69enbHCRUQkwk5qMysC/hv4aoq3VwJj3P1w4CvAb8xsaM9G7n6zu9e5e111dXV2CxYRKTDZnKxvOTA6ab0m3NZtCHAw8ISZAewLzDCz09y9EdgK4O7zzOw1YH+gMYv1imTHBalOkEX6v2yeQcwFJphZwsxKgenAjO433X2duw9391p3rwVmAae5e6OZVYed3JjZOGACsCSLtYqISA9ZO4Nw9w4zuwx4CIgBt7n7QjO7Dmh09xm72P044Dozawe6gEvcvSVbtYqIyLtl9XkQ7j4TmNlj29U7aXt80uv7gPuyWZuIiOya7qQWEZGUFBAiIpKSAkJERFLSM6lFCpSelS27o4AQKVS6P0N2w9w96hoyoq6uzhsbdR+diEhvmNk8d69L9Z76IEREJCUFhIiIpKSAEBGRlBQQIiKSkgJCRERSUkCIiEhKCggREUlJASEiIikpIEREJKW8uZPazFYDr+/FIYYD72SonExSXb2junpHdfVOPtY11t2rU72RNwGxt8yscWe3m0dJdfWO6uod1dU7hVaXLjGJiEhKCggREUlJAbHdzVEXsBOqq3dUV++ort4pqLrUByEiIinpDEJERFJSQIiISEoKiJCZ/YeZLTez+eFyStQ19WRmXzUzN7PhUdcCYGbXm9kL4d/Xw2Y2MuqaAMzse2b2cljb/WY2LOqaAMzsE2a20My6zCzSoZJmNs3MXjGzxWZ2RZS1JDOz28xslZktiLqWZGY22sweN7NF4f+HX4q6JgAzKzezOWb2fFjXtZk8vgJiRz9w90nhMjPqYpKZ2WhgKvBG1LUk+Z67H+ruk4AHgKujLij0CHCwux8KvApcGXE93RYAHweeirIIM4sBNwEfAiYCZ5vZxChrSvILYFrURaTQAXzV3ScCRwKX9pO/s63AB9z9MGASMM3MjszUwRUQueMHwNeBfjOqwN3XJ60Oop/U5u4Pu3tHuDoLqImynm7u/pK7vxJ1HUA9sNjdl7h7G3A3cHrENQHg7k8BLVHX0ZO7r3T3Z8PXG4CXgFHRVgUe2BiuloRLxv4dKiB2dFl4WeI2M6uMuphuZnY6sNzdn4+6lp7M7AYzexP4NP3nDCLZhcCfoy6inxkFvJm03kQ/+LLLFWZWCxwOzI62koCZxcxsPrAKeMTdM1ZXcaYOlAvM7K/Avineugr4CXA9QfpeD3yf4MulP9T2DYLLS31uV3W5+x/d/SrgKjO7ErgMuKY/1BW2uYrg0sCdfVFTunVJ7jKzwcB9wJd7nEFHxt07gUlhX9v9Znawu2ekD6egAsLdP5hOOzP7OcE19T6zs9rM7BAgATxvZhBcLnnWzOrd/a2o6krhTmAmfRQQu6vLzM4HPgyc6H14s08v/r6itBwYnbReE26TXTCzEoJwuNPdfx91PT25+1oze5ygDycjAaFLTCEz2y9p9WNk6C94b7n7i+4+wt1r3b2W4HLAEX0RDrtjZhOSVk8HXo6qlmRmNo2gv+Y0d98cdT390FxggpklzKwUmA7MiLimfs2C385uBV5y9/+Oup5uZlbdPUrPzAYAJ5HBf4e6kzpkZr8iGAXgwDLg8+6+MtKiUjCzZUCdu0c+5bCZ3QccAHQRTLV+ibtH/puomS0GyoDmcNMsd78kwpIAMLOPAT8CqoG1wHx3PzmiWk4BfgjEgNvc/YYo6ujJzO4CjieYvvpt4Bp3vzXSogAzOwZ4GniR4L93gG9EPdrRzA4F7iD4/7EIuNfdr8vY8RUQIiKSii4xiYhISgoIERFJSQEhIiIpKSBERCQlBYSIiKSkgBDpBTPbuPtWu9z/d2Y2Lnw92Mx+Zmavmdk8M3vCzBrMrNTMnjKzgrqRVfofBYRIHzGzg4CYuy8JN91CMDHdBHefDFwADA8n0HsU+GQ0lYoEFBAie8AC3zOzBWb2opl9MtxeZGY/Dp9H8YiZzTSzM8PdPg10zxM1HmgAvunuXQDuvtTdHwzb/iFsLxIZncKK7JmPE9x5fxjBXb9zzewp4GigluA5CyMIpoW+LdznaOCu8PVBBHdSd+7k+AuAKVmpXCRNOoMQ2TPHAHe5e6e7vw08SfCFfgzwW3fvCufLejxpn/2A1ekcPAyONjMbkuG6RdKmgBDpO1uA8vD1QuCw8OluO1MGtGa9KpGdUECI7JmngU+GD2upBo4D5gDPAGeEfRH7EEw81+0l4D0A7v4a0AhcG84UipnVmtmp4esq4B13b++rP5BITwoIkT1zP/AC8DzwGPD18JLSfQRTsi8Cfg08C6wL93mQHQPjImAfYLGZLSB4HvOq8L0TwvYikdFsriIZZmaD3X1jeBYwBzja3d8K5+t/PFzfWed09zF+D1zh7q/2QckiKWkUk0jmPRA+xKUUuL774U7uvsXMriF4/vMbO9s5fIjPHxQOEjWdQYiISErqgxARkZQUECIikpICQkREUlJAiIhISgoIERFJ6f8DCO/yHt4pjUAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# cv_results_保存了模型在整个参数搜索空间下的结果\n",
    "# mean_train_score:对于训练集，模型的交叉验证集的平均score\n",
    "# std_train_score:对于训练集，模型的交叉验证集的score标准差\n",
    "# mean_test_score:对于测试集，模型的交叉验证集的平均score\n",
    "# std_test_score:对于测试集，模型的交叉验证集的平均score\n",
    "# score就是之前模型设置的scoring，也就是neg_log_loss t\n",
    "rain_means_L1 = grid_L1.cv_results_['mean_train_score'] \n",
    "train_stds_L1 = grid_L1.cv_results_['std_train_score'] \n",
    "test_means_L1 = grid_L1.cv_results_['mean_test_score'] \n",
    "test_stds_L1 = grid_L1.cv_results_['std_test_score']\n",
    "train_means_L2 = grid_L2.cv_results_['mean_train_score']\n",
    "train_stds_L2 = grid_L2.cv_results_['std_train_score']\n",
    "test_means_L2 = grid_L2.cv_results_['mean_test_score']\n",
    "test_stds_L2 = grid_L2.cv_results_['std_test_score']\n",
    "# 生成Cs的log10值 \n",
    "x_axis = np.log10(Cs)\n",
    "# 画出误差棒图\n",
    "plt.figure()\n",
    "plt.errorbar(x_axis, -test_means_L1, yerr=test_stds_L1, label='l1') \n",
    "plt.errorbar(x_axis, -test_means_L2, yerr=test_stds_L2, label='l2')\n",
    "plt.legend()\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('logloss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "# LogisticRegressionCV + L1正则模型，预测训练集和测试集的结果 \n",
    "y_train_pred = lrcv_L1.predict(X_train)\n",
    "y_test_pred = lrcv_L1.predict(X_test)\n",
    "acc_train_lrcv_L1 = accuracy_score(y_train_pred, y_train) \n",
    "acc_test_lrcv_L1 = accuracy_score(y_test_pred, y_test)\n",
    "# LogisticRegressionCV + L2正则模型，预测训练集和测试集的结果 \n",
    "y_train_pred = lrcv_L2.predict(X_train)\n",
    "y_test_pred = lrcv_L2.predict(X_test)\n",
    "acc_train_lrcv_L2 = accuracy_score(y_train_pred, y_train) \n",
    "acc_test_lrcv_L2 = accuracy_score(y_test_pred, y_test)\n",
    "# GCV + L1正则模型，预测训练集和测试集的结果\n",
    "y_train_pred = grid_L1.predict(X_train)\n",
    "y_test_pred = grid_L1.predict(X_test)\n",
    "acc_train_grid_L1 = accuracy_score(y_train_pred, y_train) \n",
    "acc_test_grid_L1 = accuracy_score(y_test_pred, y_test)\n",
    "# LogisticRegressionCV + L2正则模型，预测训练集和测试集的结果 \n",
    "y_train_pred = grid_L2.predict(X_train)\n",
    "y_test_pred = grid_L2.predict(X_test)\n",
    "acc_train_grid_L2 = accuracy_score(y_train_pred, y_train) \n",
    "acc_test_grid_L2 = accuracy_score(y_test_pred, y_test)\n",
    "index = [\"LRCV+L1\", \"LRCV+L2\", \"GSCV+L1\", \"GSCV+L2\"]\n",
    "columns = [\"Train_accuracy\", \"Test_accuracy\"]\n",
    "acc_array = np.array([[acc_train_lrcv_L1, acc_test_lrcv_L1],\n",
    "                      [acc_train_lrcv_L2, acc_test_lrcv_L2],\n",
    "                      [acc_train_grid_L1, acc_test_grid_L1],\n",
    "                      [acc_train_grid_L2, acc_test_grid_L2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Train_accuracy</th>\n",
       "      <th>Test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LRCV+L1</th>\n",
       "      <td>0.776536</td>\n",
       "      <td>0.766234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LRCV+L2</th>\n",
       "      <td>0.780261</td>\n",
       "      <td>0.761905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSCV+L1</th>\n",
       "      <td>0.776536</td>\n",
       "      <td>0.766234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSCV+L2</th>\n",
       "      <td>0.780261</td>\n",
       "      <td>0.761905</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Train_accuracy  Test_accuracy\n",
       "LRCV+L1        0.776536       0.766234\n",
       "LRCV+L2        0.780261       0.761905\n",
       "GSCV+L1        0.776536       0.766234\n",
       "GSCV+L2        0.780261       0.761905"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 将结果用pd.DataFrame表示\n",
    "pd.DataFrame(acc_array, index=index, columns=columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4791091789402917\n",
      "{'C': 1, 'penalty': 'l1'}\n"
     ]
    }
   ],
   "source": [
    "penaltys = ['l1','l2']\n",
    "Cs = [ 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression(solver='liblinear')\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=3, scoring='neg_log_loss',n_jobs = 4,)\n",
    "grid.fit(X_train,y_train)\n",
    "\n",
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
